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Like many domains of professional psychology, school psychology 
continues to struggle with the problem of distinguishing scientific 
from pseudoscientific and otherwise questionable clinical practices. 
We review evidence for the scientist-practitioner gap in school 
psychology and provide a user-friendly primer on science and scientific 
thinking for school psychologists. Specifically, we (a) outline basic 
principles of scientific thinking, (b) delineate widespread cognitive 
errors that can contribute to belief in pseudoscientific practices 
within school psychology and allied professions, (c) provide a list of 
10 key warning signs of pseudoscience, illustrated by contemporary 
examples from school psychology and allied disciplines, and (d) offer 
10 user-friendly prescriptions designed to encourage scientific thinking 
among school psychology practitioners and researchers. We argue that 
scientific thinking, although fallible, is ultimately school psychologists’ 

best safeguard against a host of errors in thinking. 
© 2011 Society for the Study of School Psychology. Published by 
Elsevier Ltd. All rights reserved. 








All of us are prone to errors in thinking. School psychologists, despite being highly intelligent and well- 


educated, are no more immune to such errors than anyone else. In school psychology, cognitive mistakes 
are especially problematic when they contribute to the adoption of pseudoscientific practices—those that 
seem to be grounded in science but are not. Such techniques may be ineffective or even harmful to school- 
children and more broadly, can lead to suboptimal assessment, intervention, or both. Therefore, it is im- 
perative that all psychologists, including school psychologists, embrace scientific thinking tools as 
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safeguards against cognitive errors that can contribute to belief in pseudoscientific techniques. In this ar- 
ticle, we hope to persuade school psychologists—both practitioners and researchers—to incorporate such 
safeguards into their everyday thinking, clinical practice, research, and teaching. For those school psychol- 
ogists who are already making regular use of these safeguards, we hope to provide them with friendly in- 
tellectual ammunition for combating resistance to science among their colleagues. 


1. The relation between science and practice in school psychology 
1.1. The scientist-practitioner gap in school psychology 


Concerns about the wide gap between science and practice in psychology, including school psychology, 
are ubiquitous and longstanding (American Psychological Association Presidential Task Force on Evidence- 
Based Practice, 2006; Heppner, Carter, Claiborn, & Brooks, 1992; Kazdin, 2008; Weisz, Sandler, Durlak, & 
Anton, 2005). This scientist-practitioner gap, as it has come to be called (Cautin, 2011; Fox, 1996; Tavris, 
2003), is marked by a striking disjunction between the scientific evidence regarding the best available as- 
sessment and treatment practices on the one hand, and what practitioners—in this case, school psycholo- 
gists—actually do in their routine practices, on the other. 

The popular term, “scientist-practitioner gap,” may imply, misleadingly in our view, that one is necessarily 
either a scientist or a practitioner, not both. To the contrary, as in clinical psychology, we can distinguish 
between two conceptually independent dimensions in school psychology: science versus non-science, 
on the one hand, and research and practice, on the other (McFall, 1991). This two-fold distinction results 
in four quadrants. One can be a scientific researcher, a nonscientific researcher, a scientific practitioner, 
or a nonscientific practitioner. 

This distinction underscores a crucial point. The concept of the scientific practitioner is not an oxymoron, 
because even school psychologists who conduct no research can in effect act as scientists in the clinical 
setting. That is, they can bring rigorous scientific thinking to bear on the evaluation of schoolchildren's 
behavioral, emotional, and learning problems. Moreover, they can be active and discerning consumers 
of the research literature in school psychology and function as scholar-practitioners (see also Trierweiler & 
Stricker, 1992) who are continually striving to ground their practices in the best available research evidence. 

Alongside their colleagues in clinical psychology (e.g., Lilienfeld, Lynn, & Lohr, 2003) and social work (Thyer & 
Pignotti, in press), some school psychologists and allied mental health professionals have called for an increase in 
reliance on evidence-based practices to help close the scientist-practitioner gap (Bradley-Johnson & Dean, 2000; 
Hoagwood & Johnson, 2003; Hunter, 2003; Kratochwill, 2006, 2007; Kratochwill & Shernoff, 2003; Kratochwill & 
Stoiber, 2000; Miller & Nickerson, 2006; Stoiber & Kratochwill, 2000; Walker, 2004). As noted by Reynolds 
(2011), effective school psychologists must learn not to merely administer evidence-based interventions but 
also to become thoughtful and discerning consumers of the research evidence in school psychology: 


It is not enough to read the literature or to attend in-service or continuing education seminars. We must 
read and listen carefully. Just because a paper is published in a peer-reviewed journal does not mean the 
science is accurate or necessarily strong (Reynolds, 2011, p. 5). 


Put in slightly different terms, all school psychologists, regardless of the setting in which they operate, 
need to develop and maintain a skill set that allows them to distinguish evidence-based from non-evidence 
based practices. Despite the crucial importance of this skill set, some domains of school psychology, like 
those of allied fields, remain characterized by popular but poorly supported techniques. 

Indeed, the problem of questionable and unvalidated claims in school psychology and allied fields, such 
as educational psychology, is hardly new. Forty-five years ago, Glass (1968) wrote of “educational Pilt- 
down men” (p. 148) infiltrating the research literature on schoolchildren with disabilities, in reference to 
the infamous early 20th century Piltdown Man hoax that duped many credulous physical anthropologists 
into accepting the discovery of a new hominid ancestor. (The Piltdown forgery turned out to consist of a 
human skull and an orangutan jaw, with teeth from a chimpanzee, hippopotamus, and elephant tossed in 
for good measure.) Referring to “insensitivity to evidence” (p. 148) and the “wishful will to believe” (p. 150) 
as key contributors to belief in educational Piltdown men, Glass offered three examples of then-popular but 
unsupported claims concerning exceptional children: (a) a published study that purported to demonstrate 
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remarkable increases in IQ (of 35 points of more) following special (but inadequately described) teaching 
practices, but that was found to be based on entirely unsupported assertions (see also Reynolds, 1987, for 
an incisive critique); (b) a widely publicized series of studies from the University of Iowa that claimed to 
show similarly remarkable increases in IQ in economically disadvantaged schoolchildren after placement in 
foster homes, but that turned out to be due largely or entirely to enhanced rapport between schoolchildren 
and psychometricians administering the IQ tests; and (c) astonishing claims of the effectiveness of “psycho- 
motor patterning,” a neurological treatment for stuttering, reading problems, cerebral palsy, and other 
disabilities that requires children to engage in bizarre movements to recapitulate the development of their 
nervous system, but that is premised on fatally flawed research (see also Singer & Lalich, 1996). As Glass 
noted, “ignorance of valid standards of empirical evidence and a stubborn will to believe that a panacea 
had been found combined to divert precious energies and financial resources down essentially blind alleys” 
(p. 148). 


1.2. Survey evidence for the scientist-practitioner gap in school psychology 


Have things improved since Glass' commentary in 1966? Sadly, the answer is not especially clear. Like 
professionals in related fields, school psychologists are interested in promoting the well-being of the people 
they serve (Shriberg, Satchwell, McArdle, & James, 2010). Nevertheless, surveys suggest that many school 
psychologists continue to underuse science to inform their clinical practice, and that many routinely 
use questionable techniques, especially in the domain of clinical assessment. 

For example, in a survey aimed at examining regional differences in the assessment practices of American 
school psychologists (N = 351), Hosp and Reschly (2002) found that practitioners reported using projective 
measures nearly as often as behavior rating scales, which were the most frequently used indices. Projective 
techniques were especially popular in the “coastal” areas of the United States, a finding consistent with the 
prevalence of psychoanalytic thinking on both coasts (Engel, 2008). A similar and more recent survey 
by Hojnoski, Morrison, Brown, and Matthews (2006) found that projective tests continued to be widely 
administered among a sample of 175 randomly selected school psychologists. Specifically, among the 
most commonly used instruments were sentence completion tests (61%), the Bender Visual-Motor Gestalt 
Test (50%), the House-Tree-Person test (44%), and the Kinetic Family Drawing Test (41%). The authors 
noted that “not only are respondents using projective techniques, they are using them for purposes for 
which they have not been validated specifically” (p. 153). These results are broadly consistent with those 
of Wilson and Reschly (1996), who found that among school psychologists, the Draw-A-Person (DAP) test, 
a popular human drawing test, was one of the three most widely used assessment measures (see also Goh 
& Fuller, 1981; Prout, 1983, for similar results). The findings of Hosp and Reschly and those of Wilson are trou- 
bling given evidence that few projective test indices are empirically supported. This is especially the case for 
the DAP (Chapman & Chapman, 1967; Smith & Dumont, 1995) and other human figure drawing tests, which 
are of doubtful validity for detecting personality traits and psychopathological symptoms (Dawes, 1994; 
Lilienfeld, Wood, & Garb, 2000; Motta, Little, & Tobin, 1993; Wood, Nezworski, Lilienfeld, & Garb, 2003). 

Investigations of school psychologists’ familiarity with and use of empirically-supported intervention 
practices also offer at least some reason for concern. For example, a survey of a large sample (N = 496) 
of practicing school psychologists revealed that 56% of respondents felt that their “knowledge of scientific, 
research-based reading interventions for students with reading problems” was either “low” or “moderate- 
ly low” (Nelson & Machek, 2007, p. 317). Although this finding may not be particularly worrisome if these 
same school psychologists are not choosing which reading interventions to utilize, it may nonetheless be 
problematic when they are asked to provide referrals (see also Cochrane & Laux, 2008, for evidence re- 
garding many school psychologists’ lack of monitoring of treatment integrity, and O'Connor & Kratochwill, 
1999, for data on school psychologists’ use of self-help materials as psychotherapeutic adjuncts). Bramlett, 
Murphy, Johnson, Wallingsford, and Hall (2002) found that of the 370 members of the National Associa- 
tion of School Psychologists (NASP) who responded to their survey, 83% reported relying on “personal 
experiences” to inform their intervention practices (p. 330). By itself, this statistic may not be trouble- 
some given that personal experiences can be helpful in hypothesis generation in clinical and research 
contexts (Davison & Lazarus, 2007). Nevertheless, in response to the same question, only 62% of partic- 
ipants reported using reference books and only 47% reported using journal articles. Nevertheless, Bram- 
lett and colleagues did not specify whether respondents fall into one or both of these groups (i.e., Are the 
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same school psychologists who reported using journal articles the same ones who use reference books?), 
nor did they specify whether these journal articles or reference books were based on sound research. 
Even so, the fact that approximately half of participants reported not using journal articles to inform 
their clinical interventions is worrisome. 

In aggregate, the extant literature on school psychologists’ use of evidence-based practices is relatively 
small and methodologically limited. Nevertheless, the research that exists raises cause for concern and 
suggests that many school psychologists are not basing their practices on well validated techniques, particu- 
larly with respect to assessment. More research concerning school psychologists’ use of interventions is clear- 
ly warranted. 


1.3. Pseudoscience in school psychology 


As this collection of surveys suggests, a substantial number of school psychologists appear not to base their 
practices on solid science. This fact appears to be especially true in the domain of assessment. Moreover, there 
is ample reason for unease regarding the use of questionable and even pseudoscientific practices in school 
psychology. But what is pseudoscience? 

As we will later discuss in more detail, pseudoscience is “false” or “sham” science that “is not characterized 
by carefully controlled studies that result in publicly verifiable knowledge” (Lawson, 2007, p. 4). In somewhat 
different terms, we can think of pseudoscientific practices as those that “possess the superficial appearance of 
science but lack its substance” (Lilienfeld & Landfield, 2008, p. 1216; see also Lilienfeld, 1998; Ruscio, 2006). In 
other words, they are imposters of science, although their proponents are rarely deliberate or conscious 
charlatans. By definition, pseudoscientific practices lack adequate empirical support to back up their extrav- 
agant claims. In some cases, they may even cause harm (Lilienfeld, 2007). It is worth noting that pseudosci- 
entific practices are not necessarily entirely invalid or ineffective, but the assertions associated with these 
practices greatly outstrip the available scientific evidence. Indeed, a broad swath of popular pseudoscientific 
assessment and intervention strategies are germane to school psychology, including techniques associated 
with learning disorders, autism, attention deficit disorders, and conduct problems (e.g., see Lilienfeld et al., 
2003; Stanovich, 2005; Thyer & Pignotti, in press, for selected examples). We will discuss a number of 
these approaches later in the article. 


1.4. Widespread myths in school psychology 


Like most or all domains of psychology (Lilienfeld, Lynn, Ruscio, & Beyerstein, 2010), school psychology 
also has more than its share of myths: widely accepted claims that are not buttressed by compelling evidence. 
Indeed, in 1997 a special issue of School Psychology Review was devoted to prominent myths in school 
psychology, including misconceptions regarding reading and mathematics disabilities, the treatment utility 
of assessment, the treatment of attention deficit/hyperactivity disorder (ADHD), and school-based suicide 
prevention programs. Below, we offer a highly selective (and far from exhaustive) sampling of a variety of 
myths relevant to school psychology (see Lilienfeld et al., 2010; Miller & Sawka-Miller, 2011, for other 
examples). We discuss several other popular but poorly supported beliefs, such as the purported effective- 
ness of facilitated communication for autism, the notion that certain people are “left-brained” and others 
“right-brained,” and the claim that matching students’ learning styles with teachers' teaching styles boosts 
their learning, later in the article. 


* Self-esteem: Many psychologists assume that low self-esteem is highly related to maladjustment and 
poor school achievement, and many schools provide programs designed to foster self-esteem. Yet 
studies show that although self-esteem is indeed negatively correlated with depression, this correlation 
is only modest in magnitude; moreover, self-esteem is minimally related to interpersonal success and 
substance use (Baumeister, Campbell, Krueger, & Vohs, 2003; Dawes, 1994; but see Swann, Chang- 
Schneider, & Larsen McClarty, 2007, for a somewhat different view). Moreover, most research suggests 
that although low self-esteem is correlated with poor school achievement, it is not causally related to 
it (Mercer, 2010). 

DARE programs: Drug Abuse Resistance and Education (DARE) programs, in which uniformed and spe- 
cially trained police officers teach children about the dangers of substance use, are widely used in schools 
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in America and around the world; in the U.S., for example, DARE is implemented in approximately 80% of 

school districts. Nevertheless, controlled research consistently shows that DARE is ineffective in reducing 

risk for drug use (Lynam et al., 1999). 

Subtest scatter: Numerous practitioners rely heavily on subtest scatter (differences among subtests of 

omnibus tests of mental abilities, such as the Wechsler Intelligence Scale for Children, or WISC) to 

draw inferences regarding children's specific cognitive deficits. For example, many believe that the 

“ACID” profile (low scores on the WISC Arithmetic, Coding, Information, and Digit Span subtests) is as- 

sociated with a markedly higher risk of learning disabilities (Vargo, Grosser, & Spafford, 1995). Yet re- 

search shows that subtest scatter accounts for minimal, if any, variance above and beyond general 

cognitive abilities for predicting academic achievement and learning problems (Watkins, 2003). 

Late-talking children: A number of clinicians, including speech therapists, believe that late-talking chil- 

dren are at markedly elevated risk for autism. Yet data suggest that the substantial majority of late- 

talking children later develop normally, both socially and intellectually (Sowell, 1997). 

School-based suicide-prevention programs: School-based suicide intervention programs for adolescents 

are widespread and increasingly popular in many school districts. Yet controlled studies reveal surpris- 

ingly little research evidence for their efficacy and even raise the possibility that such programs are 
occasionally iatrogenic (Mazza, 1997). Recent reviews of suicide prevention programs focused on 
targeting broader audiences underscore the need for additional research before firm conclusions can 

be drawn regarding their efficacy (Mann & Currier, 2011; Rodgers, Sudak, Silverman, & Litts, 2007). 

Whole-word learning: Many teachers have argued for whole-word learning approaches (based on the 

“look-say” method) to teach children to read; programs based on these approaches continue to be wide- 

ly used. Yet research shows that more traditional reading approaches based on phonics tend to be more 

effective than whole-word instruction approaches, although phonics approaches may be helpful supple- 

ments to whole-word approaches (Rayner, Foorman, Perfetti, Pesetsky, & Seidenberg, 2001). 

Discovery learning: A widely accepted “truism” in the science and mathematics education community 

is that discovery-based approaches, in which children must uncover underlying principles (e.g., of 

the basic techniques of experimental design) on their own, results in deeper and more enduring 
knowledge than direct instruction approaches, in which children are simply told what these princi- 
ples are. Yet most research suggests that although discovery learning approaches often have some 
role to play as adjunctive educational techniques, they are generally less effective in promoting learn- 
ing of scientific and mathematical principles than are direct instruction approaches (Klahr & Nigam, 

2004). 

Dyslexia and letter reversal: A widely held belief is that the defining feature of dyslexia is the tendency to 

reverse letters, either by seeing certain letters themselves backwards (e.g., seeing the letter “b” and “d”) 

or by reversing letters within words (e.g., seeing the word “bed” as “deb”). Yet research shows that al- 

though letter reversals are generally more common among children with dyslexia than among other 
children, they are hardly key diagnostic indicators of the condition. Many children with dyslexia do 
not display letter reversals, and many non-dyslexic children aged 6 or younger display such reversals 

(Cassar, Treiman, Moats, Pollo, & Kessler, 2005; Shaywitz, 1996). 

e Autism epidemic: Much of the news media has promulgated the claim that the rates of autism have in- 
creased dramatically since the early 1990s. Yet there are serious reasons to question this assertion, 
because the diagnostic criteria for autism have become considerably less stringent over time and be- 
cause the reported increases in prevalence derive not from population-based studies that rely on 
standardized diagnostic criteria for autism, but from school-based studies in which educational dis- 
tricts report the number of children diagnosed with autism (Gernsbacher, Dawson, & Goldsmith, 
2005). Indeed, in one study in the United Kingdom, the rates of autism remained unchanged from 
1992 to 1998 when the identical diagnostic criteria were used at both assessments (Chakrabarti & 
Fombonne, 2005). 


School psychology myths are important, because they can produce either direct or indirect harm (see 
also Lilienfeld et al., 2010). For example, school psychologists who rely heavily on intelligence test subtest 
scatter to diagnose learning disabilities will often misdiagnose these problems, and school psychologists 
who assist in developing unvalidated suicide intervention programs may waste valuable time, energy, 
and money on ineffective techniques for potentially life-threatening student problems (to which 
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economists refer as “opportunity costs”). Thus, we believe that school psychologists in training should be 
introduced to these and other prevalent myths during the course of their graduate education. 


1.5. Goals of this article 


Our overarching goals in this article are fourfold. First, we outline several basic principles of scientific 
thinking, with particular reference to school psychology. We argue that science and scientific thinking 
are valuable, indeed essential, safeguards against the often seductive charms of pseudoscience. They can 
also help us to identify superficially compelling myths and misconceptions pertinent to school psychology. 
Second, we delineate widespread cognitive errors that can impede the accurate evaluation of clinical prac- 
tices and can contribute to the adoption of pseudoscientific practices and the embrace of misconceptions, 
especially in school psychology. In addition, we discuss potential strategies for minimizing these cognitive 
errors. Third, we provide a set of user-friendly criteria for distinguishing science from pseudoscience, 
again with particular reference to practices in school psychology. Fourth, we conclude with a set of con- 
crete prescriptions to school psychology researchers and practitioners for incorporating scientific thinking 
into their everyday work. 

Across the four major sections of our article, our overarching goal is to equip school psychologists and 
psychologists in allied domains with scientific thinking skills that can better allow them to navigate the 
often bewildering mix of well-supported, poorly-supported, and equivocally-supported practices that 
comprise their fields. In this way, readers should emerge with tangible tools for sorting the wheat from 
the chaff in school psychology and for distinguishing science from pseudoscience across all domains of 
psychology. 


2. Basic principles of scientific thinking: guidelines for school psychologists 
2.1. Science: a rough definition 


We have already offered a thumbnail definition of pseudoscience. But what is science? Philosophers of 
science have long debated the answer to this deceptively complex question. Although we do not intend to 
offer a conclusive answer here, we offer some helpful guidelines. 

Whatever science may be, it is clear what it is not. Science is not a body of accumulated facts and fig- 
ures, nor is it a monolithic truth-gathering device that is identical across diverse disciplines. Although 
“the” scientific method is often introduced in high school biology, chemistry, or physics classes as a single 
technique for helping us understand the state of nature, each scientific discipline is associated with its own 
set of safeguards against error. Hence, the concept of “the” scientific method is almost surely a myth 
(Bauer, 1994) because there are multiple scientific methods, each tailored to answer different kinds of 
questions. Diverse scientific fields, including psychology, chemistry, biology, and astrophysics, use vastly 
different methods for minimizing errors and arriving at a closer approximation of the truth. Even within 
psychology, the research methods used by clinical, counseling, and school psychologists (e.g., correlational 
designs to examine the predictive validity of intelligence for academic performance), for example, differ 
sharply from those used by perceptual psychologists (e.g., psychophysical methods to investigate partici- 
pants’ ability to distinguish between two similar visual stimuli). 

Yet despite their marked surface differences, the scientific methodologies used by different fields outside 
of and within psychology share one key feature: a commitment to rooting out errors in our web of beliefs 
(O'Donohue, Fowler, & Lilienfeld, 2007). Ultimately, as most philosophers conceive of it, the systematic 
approach to knowledge, rather than the characteristics of knowledge per se, renders a discipline scientific. 
Specifically, as Nobel-prize winning physicist Richard Feynman (1985) argued, the essence of science is 
bending over backwards to prove ourselves wrong. As Feynman noted, we should be continually striving 
to find ways of subjecting our most cherished beliefs to rigorous testing and to changing our minds when 
the evidence compels us (O'Donohue, Fowler, & Lilienfeld, 2007). 

As astronomer and science writer Carl Sagan (1995a) observed, science is first and foremost a way of 
thinking, a mindset of doubt coupled with an open-mindedness to new claims. Moreover, as space engi- 
neer James Oberg (see also Sagan, 1995b) reminded us, we must strike a healthy balance between 
doubt and open-mindedness: this balance is the approach often termed skepticism. Skepticism, in contrast 
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to cynicism, implies a willingness to consider novel claims but also an insistence on evidence to support 
them. Cynicism is marked by dismissiveness toward new and untested claims, and in many respects is 
just as problematic as excessive open-mindedness (Beyerstein, 1995). In Sagan's (1995a) eloquent words, 


Science involves a seemingly self-contradictory mix of attitudes. On the one hand, it requires an al- 
most complete openness to all ideas, no matter how bizarre and weird they sound, a propensity to 
wonder...But at the same time, science requires the most vigorous and uncompromising skepti- 
cism, because the vast majority of ideas are simply wrong, and the only way you can distinguish 
the right from the wrong, the wheat from the chaff, is by critical experiment and analysis. Too 
much openness and you accept every notion, idea, and hypothesis—which is tantamount to knowing 
nothing. Too much skepticism—especially rejection of new ideas before they are adequately tested— 
and you're not only unpleasantly grumpy, but also closed to the advance of science. A judicious mix 
is what we need (p. 30). 


2.2. Science versus common sense 


In sharp contrast to writers who have argued that science is little more than formalized common sense 
(e.g., Huxley, 1863; see also Horgan, 2005), we side with those (e.g., Cromer, 1993; McCauley, 2000; Wolpert, 
1993) who have contended that the essence of science is “uncommon sense.” That is, science is unnatural, 
because it often requires us to override our gut hunches and intuitions about the natural world, including 
the psychological world. 

Perhaps most centrally, science is a potent antidote against naive realism (Ross & Ward, 1996), the 
tempting but erroneous assumption that the world is exactly as we see it. A host of sayings in our language 
attest to the hold of naive realism on our everyday thinking: “Seeing is believing,” “I'll believe it when I see 
it,” “I saw it with my own eyes.” Indeed, naive realism is arguably the wellspring of many of the most spec- 
tacularly wrongheaded scientific theories in history. For most of human history, people assumed that the 
earth was the center of the solar system, if not the universe, because their raw perceptions—and their 
naive realism—assured them that this was the case. After all, each day, we seemingly remain planted on 
terra firma as the sun paints a huge arc around the sky from dawn to dusk (Lilienfeld et al., 2010). Thus, 
from a phenomenological perspective, we do indeed seem to be at the center of things, although this perspec- 
tive is of course misleading. 

Similarly, when a school psychologist conducts an intervention to alter a child's problematic classroom 
behavior and observes improvement following that intervention, he or she may assume that the intervention 
was responsible for the change. That conclusion would be understandable, as it dovetails with naive realism: 
“I saw the change with my own eyes.” But this inference would be erroneous, as it reflects the logical error of 
“post hoc, ergo propter hoc” (after this, therefore because of this). Specifically, the conclusion neglects to 
consider a plethora of rival explanations for the improvement, including naturally occurring changes (e.g., 
spontaneous remission of symptoms), regression to the mean (the tendency of extreme scores to become 
less extreme following re-testing), placebo effects (improvement resulting from the mere expectation of 
improvement), multiple treatment interference (attributing improvement to a specific treatment when in 
fact it is due to a co-occurring intervention), novelty effects (the tendency of participants to react differently 
to any new intervention), and other artifacts (Beyerstein, 1999; Lilienfeld, Lohr, & Olatunji, 2008). Instead, to 
evaluate whether the intervention worked, school psychologists must avail themselves of scientific methods, 
such as randomized controlled trials (Chambless & Ollendick, 2001) and systematic within-subject designs 
(Kazdin, 2010), which help to exclude most of these sources of error. 

At least some of the reluctance to embrace evidence-based practice in school psychology is therefore 
understandable, and it largely reflects a preference for intuitive, “commonsense” reasoning over scientific 
reasoning. The former form of reasoning, we contend, comes more naturally and automatically to all of us. 
Scientific reasoning, in contrast, is unnatural: it must be learned, practiced, and maintained as a habit of 
mind (Cromer, 1993; Watkins, 2009). Indeed, recent work in developmental psychology suggests that 
young children display certain deep-seated impediments to scientific thinking, such as a tendency to 
infer teleology (i.e., purpose) and causation even when it is absent (Bloom & Weisberg, 2007). Simply 
put, scientific thinking is not a natural way of conceptualizing the world (McCauley, 2000). Hence, it is 


14 S.O. Lilienfeld et al. / Journal of School Psychology 50 (2012) 7-36 


hardly surprising that scientific thinking is a relatively recent development in human history, originating 
in incipient form in Ancient Greece and not emerging full-fledged until the European Enlightenment in the 
late 17th century (Lilienfeld, 2011). 

When viewed from this perspective, some school psychologists’ resistance to evidence-based assessment 
and treatment approaches is not terribly surprising. Most of us are inclined to trust our intuitions and informal 
observations, which tend to be personal, tangible, and subjectively immediate. We also tend to privilege them 
over scientific findings, which tend to be impersonal, abstract, and subjectively distant. Hence, when school 
psychologists’ intuitions regarding the validity of an assessment technique or the efficacy of a psychological 
intervention conflict strongly with well-replicated scientific evidence, they may often side with their intui- 
tions. Yet the lengthy history of errors in medicine, psychology, and other disciplines teaches us that clinical 
intuitions, although certainly helpful in many circumstances, are fallible (Dawes, Faust, & Meehl, 1989). Many 
historians of medicine have argued that prior to about 1890, the history of medicine was largely the history of 
the placebo effect and that most procedures practiced by physicians were useless, harmful, or at best, no more 
useful than a placebo. For centuries, physicians “knew”—based on their clinical experience—that such 
treatments as blistering, purging, and blood-letting were effective, even though we now recognize that 
this “knowledge” was woefully mistaken (Grove & Meehl, 1996). 

Still, it is worth noting that intuitions are hardly useless despite their fallibility (Kahneman & Klein, 
2009; Myers, 2002). In particular, intuitions can sometimes be extremely helpful in what philosophers 
of science sometimes term the “context of discovery” (Meehl, 1990; Reichenbach, 1938)—that is, hypothesis 
generation. For example, an experienced school psychologist who has a strong hunch that a child has been 
sexually abused should in most cases be certain to check out this hunch, while bearing in mind the hazards 
of confirmation bias (see below). But in the “context of justification,” that is, systematic hypothesis testing, 
an overreliance on intuitive thinking can be dangerous, even disastrous. That is because our intuitions, 
which rely on quick, associative thinking rather than slow, analytical thinking, are not generally well suited 
to addressing scientific questions (Kahneman, 2011). For example, intuitive, “gut level” thinking is not 
typically geared to ruling out rival hypotheses in school psychology settings, such as hypotheses concerning 
whether a child with ADHD's improvement following a psychological intervention is due to the intervention 
itself or to a host of other factors (e.g., placebo effects, regression to the mean, maturation). 

The bottom-line message here is not to reflexively ignore our clinical intuitions, as our hunches and 
subjective clinical observations (a) are sometimes correct and (b) can sometimes be helpful in generating 
fruitful hypotheses to later be tested in rigorous investigations. Instead, we must treat our clinical intuitions 
with healthy skepticism and be prepared to modify or even discard them when well-controlled scientific 
evidence repeatedly contradicts them. In addition, school psychologists should continually strive to put 
their powerful intuitions to systematic tests in both clinical and research settings. In some cases, these 
intuitions will turn out to be accurate; in others, they will turn out to be erroneous. 

Only scientific methods—safeguards against error—can ultimately tell us whether our intuitions are 
correct or incorrect. If the history of science has taught us anything, it is that subjective certainty is an 
untrustworthy guide to the veridicality of scientific claims (see Gardner, 1957; Hyman, 2001; for a number 
of classic examples). We should listen to our intuitions, but we should not be imprisoned by them. 


3. Commonplace cognitive errors relevant to school psychology 


School psychologists, like all practicing psychologists, must remain vigilant against a host of errors in 
thinking that can result in suboptimal practice (see Croskerry, 2003; Crumlish & Kelly, 2009; Gambrill, 
2005; Watkins, 2009, for comprehensive descriptions of these and other cognitive errors). These errors 
may also contribute to the uncritical acceptance of questionable practices, especially those that superficially 
appear to be scientific but are not (Gilovich, 1991). Like visual illusions, these errors, which we can think of 
as cognitive illusions (Piatelli-Palmarini, 1994; Pohl, 2005), tend to be subjectively compelling: They do not 
seem to us to be errors at all, even though they are. Moreover, like visual illusions, we may never be able to 
overcome them fully, because they are byproducts of basically adaptive psychological tendencies, especially 
our tendency to extract meaning from our worlds even when such meaning is absent (Shermer, 2011). 
Putting it a bit differently, most errors in reasoning are probably cut from the same cloth as effective rea- 
soning. The good news is that by becoming aware of these cognitive errors, we may be able to develop 
thinking skills or habits of mind to overcome them. 
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Incidentally, there is no reason to believe that researchers, ourselves included, are immune from any of the 
cognitive errors we describe below. To the contrary, at least some data suggest that scientific researchers may 
be even more prone to certain errors in thinking, such as confirmation bias (see the paragraph that follows), 
than are nonscientists (Mahoney & DeMonbreun, 1977). The key difference between successful and unsuc- 
cessful scientists is not that the former are less susceptible to errors in thinking than the latter. Instead, the 
key difference is that the former typically avail themselves of the safeguards of scientific methodology, 
whereas the latter do not. This key difference in mindset, we propose, also separates successful practitioners 
from unsuccessful practitioners in school psychology and related fields. Ironically, good practitioners are 
typically those who are most cognizant of their propensity toward mistakes, because they can make con- 
certed efforts to minimize them. 

Before proceeding further, it is worth noting that cognitive errors are largely independent of overall 
intelligence (Stanovich, 2009) and hence should not be attributed to low cognitive ability. Indeed, the 
history of science is replete with vivid examples of highly intelligent scientists, even Nobel-prize winners, 
acting in foolish and irrational ways (Hyman, 2001; Shermer, 2002). Two-time Nobel-prize winner 
Linus Pauling, for example, was convinced that large doses of Vitamin C could cure cancer (Pauling, 1980), 
despite overwhelming evidence to the contrary. Such observations dovetail with our earlier point that 
scientific thinking does not come naturally to the human mind: highly intelligent and even brilliant 
individuals can be prone to serious lapses in such thinking. This is why we need science. 

Here we provide a selective list of cognitive errors, with a particular focus on seven classes of errors 
that are especially pertinent to the everyday work of the school psychologist (see also Wedding & Faust, 
1989, for cognitive errors relevant to neuropsychological assessment). Table 1 presents a list of these 
seven major classes of cognitive errors, along with a number of other errors that are important for school psy- 
chologists to consider and avoid (see also Watkins, 2009, for an excellent overview). 


3.1. Confirmation bias 


As we noted earlier, science is a toolbox of techniques designed to minimize human error. More formally, 
we can regard science as a formalized set of tools for overcoming confirmation bias (Nickerson, 1998), the 


Table 1 
Widespread cognitive errors and biases relevant to school psychologists. 





Error Description 


Naive realism 
Confirmation bias 


Belief that the world is precisely as we see it 
Tendency to seek out evidence consistent with our beliefs, and deny, dismiss, or distort 
evidence that is not 


Premature closure 

Belief perseverance 
Illusory correlation 
Hindsight bias 
Groupthink 

Overreliance on heuristics 
Availability heuristic 
Anchoring heuristic 
Affect heuristic 
Representativeness heuristic 
Base rate neglect 

Bias blind spot 
Unpacking* 

Aggregate bias* 

Search satisfying* 


Diagnostic overshadowing* 
Diagnosis momentum? 


Psych out error? 
Pathology bias* 


Coming to a premature conclusion before adequate evidence is available 
Tendency to cling to beliefs despite repeated contradictory evidence 

Tendency to perceive statistical associations that are objectively absent 

Error of perceiving events as more predictable after they have occurred 
Preoccupation with group unanimity that impedes critical evaluation of an issue 
Tendency to place too much weight on mental shortcuts and rules of thumb 
Judging the probability of an event by the ease with which it comes to mind 
Tendency to be unduly influenced by initial information 

Judging the validity of an idea by the emotional reaction it elicits in us 

Judging the probability of an event by its similarity to a prototype 

Neglecting or ignoring the prevalence of a characteristic in the population 
Tendency to see ourselves as immune to biases to which others are prone 
Failure to elicit all relevant diagnostic information about an individual 
Assumption that group data are irrelevant to a given individual or client 
Tendency to end one's search for other diagnostic possibilities once one has 
arrived at an initial diagnostic conclusion 

Tendency for a dramatic, salient diagnosis (e.g., major depression) to lead clinicians to 
overlook less obvious diagnoses (e.g., learning disability) 

Tendency to uncritically accept previous diagnoses of the same individual 
Neglecting the possibility that a behavioral problem is medical, not psychological, in origin 
Tendency to overpathologize relatively normative behavior 





* Note. Error or bias not discussed in the text. 
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deeply ingrained tendency to seek out evidence consistent with our hypotheses and to deny, dismiss, or 
distort evidence that is not (Tavris & Aronson, 2007). We can sum up confirmation bias in five words: “Seek 
and ye shall find” (Lilienfeld, Lynn, Namy, & Woolf, 2011). As described eloquently by Baron (1994), confir- 
mation bias “makes us into lawyers, hired by our own earlier views to defend them against all accusations 
rather than detectives seeking the truth itself” (p. 218; see also Watkins, 2009). To a large extent, scientific 
thinking is designed to minimize confirmation bias, especially the tendency for investigators to find evidence 
for their preferred hypotheses. Richard Feynman (1985) put it well when he described science as a set of 
methods developed to minimize the chances that we can be fooled—and fool ourselves. 

In the domain of school psychology, confirmation bias can manifest itself in a host of ways. For example, a 
school psychologist may form an initial impression of a child as possessing a learning disability. She may then 
selectively seek out all evidence consistent with that diagnosis, such as anecdotal reports from school teachers 
and parents, while ignoring or minimizing evidence that is inconsistent with it, such as average performance 
on a well-validated test battery designed to detect learning disabilities. Unless school psychologists are 
perpetually on guard against their own deep-seated preconceptions, confirmation bias can easily become 
“tunnel vision,” blinding them to evidence that calls their hunches and hypotheses into question. 

In psychological research, essential safeguards against confirmation bias include (a) randomized controlled 
trials, which help to control for the experimenter expectancy effect (i.e., the tendency for researchers’ precon- 
ceptions to unintentionally bias their recording, interpretation, or both of findings) and other biases 
(Rosenthal, 1994), (b) systematic within-subject designs, which help to control for the erroneous tendency 
of observers to attribute naturally occurring changes over time within subjects to the treatment (Moran & 
Tai, 2001), (c) longitudinal designs, which help to control for hindsight bias (Arkes, 1991), that is, perceiving 
outcomes as more predictable than we would have in advance (see section on hindsight bias that follows), 
and (d) correlational designs, which minimize the probability of illusory correlations, that is, perceiving statis- 
tical associations in their objective absence (see section on illusory correlation that follows; Chapman & 
Chapman, 1967, 1969). In regard to the last design, it is crucial that we not lose sight of the familiar statistical 
mantra that correlation does not necessarily imply causation (i.e., the fact that an association between two 
variables exists does not imply a causal association between them). 

In psychological practice, including that of school psychology, scientific practice similarly incorporates 
formal safeguards against confirmation bias and allied errors. For example, standardization of aptitude test 
administration and scoring minimize—although do not eliminate—the likelihood that a school psychologist 
who believes that a child is intellectually gifted will inadvertently give that child “a break” when scoring 
certain items, especially those that he or she barely missed. By incorporating standardized questions and 
prompts, structured and semi-structured interviews for child psychiatric disorders, such as the Kiddie- 
Schedule for Affective Disorders and Schizophrenia (KIDDIE-SADS; Puig-Antich & Chambers, 1986), similarly 
minimize the likelihood that interviewers’ initial hunches about a child's diagnosis or diagnoses will become 
self-fulfilling prophecies. 

Confirmation bias is associated with two closely related errors that are relevant to school psychology. 
Premature closure (Gauron & Dickinson, 1969) is the tendency to arrive at confident diagnostic judgments 
before adequate data are available. For example, a school psychologist might quickly leap to the conclusion 
that a child has attention-deficit/hyperactivity disorder (ADHD) on the basis of an initial classroom obser- 
vation that he is jumpy and fidgety. This conclusion would typically be problematic because many normal 
children, including intellectually gifted children, display such behaviors on occasion, especially when bored or 
unstimulated (Hartnett, Nelson, & Rinn, 2004). A second related error, belief perseverance, is the tendency to 
cling to initial ideas despite repeated contrary evidence (Ross, Lepper, & Hubbard, 1975). For example, when 
presented with evidence that the child mentioned immediately above was not perceived as especially 
inattentive, impulsive, or hyperactive by his parents or by several independent teachers, the same school 
psychologist might continue to insist that this child must have ADHD. Although holding on to one's beliefs 
in the face of contrary evidence is sometimes warranted if there is good reason to doubt the quality or 
quantity of that evidence, one must be willing to abandon it when the evidence becomes convincing. 

There is no simple “cure” for confirmation bias. Nevertheless, research suggests that by forcing ourselves 
to “consider the opposite”—that is, to ask ourselves what conclusions we would have drawn had the findings 
contradicted our beliefs—we can sometimes compensate for confirmation bias by envisioning alternative 
hypotheses (Lilienfeld, Ammirati, & Landfield, 2009; Lord, Lepper, & Preston, 1984). For example, imagine a 
school psychologist who has a strong suspicion that a girl was sexually abused by her father. The psychologist 
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then comes across evidence that the child is usually shy and inhibited around male adults, including male 
teachers. He concludes that such behaviors support his suspicion of abuse. To consider the opposite, the 
school psychologist should ask himself “Would I have drawn the same conclusion if the child were unusually 
comfortable or excessively friendly around male teachers?” If the answer is a possible or probable yes, the 
teacher may become less confident of his conclusion—as he probably should. 


3.2. Illusory correlation 


As a number of psychologists (e.g., J.E. Alcock, 1981; Pinker, 1997) and science writers (e.g., Shermer, 
2002) have noted, the human mind is a pattern-seeking organ. Shermer (2011) has aptly described it as “a 
belief engine.” We tend to look for and find connections among events in our environments. In this way, 
we often find order in disorder and sense in nonsense. The adaptive significance of this tendency is unde- 
niable, as it helps us to make sense of our often bewildering worlds. 

Nevertheless, this meaning-making tendency can sometimes lead us to perceive statistical associations 
even when they are objectively absent, a remarkable phenomenon that psychologists Chapman and Chapman 
(1967, 1969) termed illusory correlation. Many school psychologists are convinced that a strong association 
exists between certain “signs” or indicators in children's human figure drawings and their propensity for 
psychological problems. For example, some school psychologists believe that physical proximity in kinetic 
family drawing tests (e.g., placing oneself near one's parents in the drawing) is a good indicator of interper- 
sonal closeness to others. Others believe that the presence of large eyes in a DAP drawing is a dependable 
indicator of suspiciousness. Yet controlled research demonstrates that these associations are largely or 
entirely illusory, as are the substantial majority of other purported associations between specific figure 
drawing signs and either personality or psychopathology (Motta et al., 1993; also see Gresham, 1993; 
Kahill, 1984; but see Holtzman, 1993, for a different view). 

Although there are probably multiple sources underlying illusory correlation, perhaps the most important 
for our purposes is the all-too-human tendency to recall our hits, and forget our misses. When a finding seem- 
ingly corroborates our expectations, we usually attend to it and remember it, an error termed the “fallacy of 
positive instances” (Coon, 2005). In contrast, when a finding violates our expectations, we more often than 
not ignore it or explain it away as a “near miss” (Gilovich, 1991). By predisposing us to weight certain occur- 
rences more than we should, this propensity toward selective perception and memory can lead us to perceive 
statistical associations that are not present as well as to perceive weak statistical correlations as strong in 
magnitude. 

By learning to attend to our misses as well as our hits, we may be able to overcome or at least minimize 
our propensities toward illusory correlation (see T. Alcock, 1981). For example, a school psychologist 
whose clinical experience leads her to believe that children who are intellectually gifted tend to have 
poor social skills may wish to force herself to keep track of all of the instances that do not fit this pattern, 
especially cases of intellectually gifted children who are socially skilled. She may also want to force herself 
to keep track of all of the cases of non-intellectually gifted children who are socially unskilled. By doing so, 
she may help to avoid the fallacy of positive instances by attending to instances that do not corroborate 
her expectations. 


3.3. Hindsight bias 


We have all learned that hindsight is 20-20, but this axiom is easy to forget in the busy world of everyday 
clinical practice. Indeed, we are all susceptible to hindsight bias—the “I knew it all along effect”—which is the 
error of perceiving events as more predictable after they have occurred compared with before they occurred 
(Fischoff, 1975; Hawkins & Hastie, 1990). Arkes, Wortmann, Saville, and Harkness (1981) presented physi- 
cians with a hypothetical case history of a patient with ambiguous physical symptoms along with four alter- 
native medical diagnoses that could explain his symptoms. Some physicians were not told which of the four 
diagnoses were correct; others were. Sure enough, the latter physicians were much more certain that they 
would have made the correct diagnosis than were the former. The knowledge of the outcome had biased 
their estimates of the certainty of their predictions. 

As Arkes et al. (1981) observed, hindsight bias can contribute to overconfidence, as well as to second 
opinions spuriously corroborating first opinions. For example, a school psychologist might learn that a 
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new child in the school who speaks up frequently in class and challenges his teachers’ authority was diag- 
nosed by a psychologist at his previous school with oppositional defiant disorder (ODD). Having learned 
this information, the school psychologist may become overly confident in her diagnosis of the child as having 
ODD and may conclude that she surely would have made the same diagnosis based on the child's behavior 
(e.g., “From practically the moment I saw him in the classroom, I could tell that he had ODD”). This 
undue confidence may then lead her to become unduly certain of her future diagnostic judgments. 

Hindsight bias is potentially problematic for reasons other than overconfidence. For example, it can 
lead school psychologists to engage in “deterministic reasoning” (Garb, 1998; Moran & Tai, 2001): pre- 
suming that an event that preceded a pathological outcome is likely to be causally related to that outcome. 
For example, the first author of this article consulted on a case in which an adopted young adolescent was 
charming and intelligent, yet dishonest, disruptive, and oppositional with adults. Perhaps largely because 
of their knowledge of the child's behavior, the school psychologist and a social worker working with the 
teenager were quick to place blame on the adoptive parents for the child's misbehavior, despite any com- 
pelling evidence that their parenting skills were suboptimal and in spite of at least some evidence to the 
contrary (e.g., that their biological children were well adjusted). Moreover, they initially ignored plausible 
competing hypotheses, such as the possibility that much of the child's behavior could reflect his genetic back- 
ground; indeed, there was ample evidence that his biological father not only had a lengthy criminal record but 
also exhibited many classic features of psychopathic personality, such as lack of guilt and empathy (see Hare, 
2003). 

Although there is no foolproof strategy for overcoming hindsight bias, at least some evidence suggests 
that generating reasons for why possibilities other than the actual outcome might have occurred can mini- 
mize such bias (Arkes, Faust, Guilmette, & Hart, 1988; but see Sanna & Schwarz, 2003, for evidence that asking 
participants to list too many reasons may occasionally backfire). For example, a school psychologist who is 
certain that a previous psychologist's diagnosis of ODD for a given child is correct may wish to generate 
reasons why alternative diagnoses (e.g.. ADHD) might instead be correct. By doing so, he may come to 
realize that the diagnosis was not so self-evident after all. 


3.4. Groupthink 


Because we humans are social beings, we are readily influenced by the opinions of others. This is not 
necessarily a net negative, as the views of other people are often worth considering. Yet our susceptibility 
to group opinion can occasionally predispose us to groupthink, a term coined by Janis (1972) based on his 
analysis of the Kennedy administration's disastrous 1961 decision to invade Cuba's Bay of Pigs. According 
to Janis, groupthink is a preoccupation with group unanimity that can impair the critical evaluation of deci- 
sions. As Janis noted, groupthink comprises a host of “symptoms,” including (a) a pressure toward 
conformity (e.g., “We really need to come to a consensus on this issue”), (b) an illusion of the group's unanim- 
ity (e.g., “We all agree, right?”), (c) an illusion of the group's correctness (e.g., “Clearly, we're on the right 
track”), (d) the presence of mindguards, that is, self-appointed members of the group who suppress dissent 
(e.g., “Are you sure you know what you're talking about?”), and (e) self-censorship, that is, the tendency to 
keep doubts about the group's decision to oneself (e.g., “This seems like a really bad decision, but I must be 
missing something, so I'll keep quiet”). Admittedly, groupthink is not invariably negative (Tyson, 1987), as 
it can sometimes lead to accurate and efficient decisions. Yet groupthink has a pronounced dark side, as it 
may lead groups, even those comprising highly intelligent and informed individuals, to make profoundly 
misguided decisions. 

Because school psychologists often work in teams of other professionals, they must remain vigilant 
against the perils of groupthink. For example, some authors have argued that groupthink may sometimes 
come into play during team decisions regarding special education disability services (e.g., VanDerHeyden, 
Witt, & Naquin, 2003; Ysseldyke, 1987). In such cases, school psychologists may feel compelled to accede 
to other members’ opinions about whether a child is entitled to disability accommodations even when 
they disagree with them. Gutkin and Nemeth (1997) noted in their review of the literature that “Multidis- 
ciplinary diagnostic teams in school settings, for example, have been found to reach inappropriate deci- 
sions based on irrelevant student data (e.g., naturally occurring pupil characteristics such as gender, 
socioeconomic status, and physical appearance), resource availability, and ‘teacher squeak’ (i.e., how potently 
a teacher complains about a referred student)” (p. 197). 
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Fortunately, as Janis (1972) observed, groupthink may be mitigated by explicitly encouraging minority 
dissent and perhaps by appointing a “devil's advocate” whose explicit role is to raise questions regarding 
group decisions (see also Gutkin & Nemeth, 1997). At least some research supports the value of minority 
dissent in encouraging diversity of thinking in teams (Nemeth, Brown, & Rogers, 2001). 


3.5. Overreliance on heuristics 


Heuristics are mental shortcuts or rules of thumb that we all use to simplify and streamline our processing 
of information (Herbert, 2010; Tversky & Kahneman, 1974). In general, heuristics are adaptive and efficient, 
as they allow us to make “fast and frugal” decisions based on relatively little information (Gigerenzer & 
Brighton, 2009). For example, the “recognition heuristic” is the guideline that “If we've heard of something, 
it's probably higher in quality” (see Goldstein & Gigerenzer, 1999). So, if we learn that an academic job 
candidate graduated from the psychology department at Harvard University, we might be inclined to 
conclude that the candidate—all else being equal—is better trained than if that candidate graduated 
from the psychology department at Nowhere Community College. 

But as this example amply illustrates, heuristics can sometimes lead us astray, as all things are not always 
equal. Indeed, some job candidates from Nowhere Community College are probably better trained than those 
from Harvard University, so recognition is not a foolproof barometer of quality. Similarly, a practitioner might 
assume that Test X is a more construct valid measure of reading disabilities than is Test Y merely because it is 
better known, but perhaps Test X is less valid for this purpose but has been marketed and promoted more 
extensively. The last several decades of psychological research have taught us that heuristics, although 
generally helpful and even necessary for functioning in everyday life, can lead to predictable errors if 
we apply them uncritically (Gilovich, Griffin, & Kahneman, 2002). Here, we briefly review four heuristics 
that are especially pertinent to school psychology (see Herbert, 2010, for a comprehensive description of 
heuristics). 

The availability heuristic is the tendency to judge the likelihood of an occurrence by the ease with which 
it comes to mind. For example, if] ask a school psychologist whether more boys than girls in his school are 
on stimulants (e.g., Ritalin) for ADHD and related disorders, he may quickly bring to mind the many cases 
of boys on stimulants and conclude—in this case probably correctly—that more boys than girls in his school 
are on stimulants. Nevertheless, the availability heuristic can sometimes lead us to judge probabilities inaccu- 
rately, especially if we accord undue weight to certain salient, but unrepresentative, examples. For instance, if 
we hear of two or three tragic shootings in various U.S. schools over the past few months, we may be tempted 
to conclude that violence in schools is of epidemic proportions, because such violence comes readily to mind. 
In this case, we would be mistaken, as most evidence actually suggests a decrease, not an increase, in school 
violence over the past decade or two (Cornell, 2006). 

The anchoring heuristic is the tendency to be influenced too heavily by initial information; once we obtain 
such information, we often do not adjust away from it sufficiently. For example, a school psychologist may 
learn mistakenly that a child's IQ is low (e.g., 75). When later informed that the IQ test was administered 
and scored incorrectly and that the child's actual IQ is closer to 100, the psychologist may not fully “wipe 
the slate clean” and disregard the initial information. Instead, the psychologist may remain anchored to the 
earlier IQ estimate and assume erroneously that the child is of below average overall cognitive ability. We 
can also think of confirmation bias and its close relatives (i.e., premature closure and belief perseverance) 
as derivatives of the anchoring heuristic, as this bias reflects a failure to modify our initial beliefs sufficiently 
following the receipt of new information. 

The affect heuristic, sometimes also referred to as “emotional reasoning,” is the tendency to use our 
emotions as guides to evaluating the veracity of an assertion. More often than not, if a claim arouses positive 
emotions in us, we may be inclined to believe it; conversely, if a claim arouses negative emotions in us, we 
may be inclined to disbelieve it. Imagine a school psychologist whose graduate school training has led him 
to believe that projective testing is helpful for detecting child sexual abuse (see Garb, Wood, & Nezworski, 
2000, for evidence to the contrary). Further, assume that he relies heavily on projective techniques in 
his everyday clinical practice and views them as broadly consistent with his psychodynamic perspective 
on personality. Upon attending a conference presentation that calls into question the use of projective 
techniques for detecting child sexual abuse, he may become angry and upset and use this negative emotional 
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reaction as an indication that the conference presenter's conclusions were mistaken. Yet becoming a good sci- 
entist requires us to set aside our emotions in favor of data, a capacity that few (if any!) of us are good at. 
The representativeness heuristic is the tendency to assume that “like goes with like.” When we use this 
heuristic, we are gauging the likelihood that an event or object is a member of a broader class by evaluating its 
similarity to a prototype. In essence, we are judging a book by its cover. Of course, sometimes books do 
resemble their covers: most people who seem very angry in fact are angry, and most people who greet 
us with a warm smile like us. Yet if used uncritically, the representativeness heuristic can lead us to commit 
systematic mistakes, because appearances can sometimes be deceiving. For example, when administering an 
IQ test to a child referred for behavioral difficulties, a school psychologist might fixate on the fact that some of 
the child's responses (e.g., idiosyncratic definitions of several vocabulary items) remind him of responses he 
has observed among a few children with autism spectrum disorders. By placing undue weight on these 
similarities, the psychologist may conclude erroneously that the child has an autism spectrum disorder. 


3.6. Using heuristics thoughtfully 


Heuristics are essential for everyday functioning, and they are also essential for the routine clinical 
practice of the school psychologist. They save time and often lead to approximately correct solutions 
(Gigerenzer, 2007). But as we have learned, they can lead to certain predictable errors if used uncritically 
(Herbert, 2010; Tversky & Kahneman, 1974). Thus, it is crucial for school psychologists to learn to question 
their initial judgments, especially when they are relevant to important clinical decisions, and actively seek 
out evidence that might call them into question. 


3.7. Base rate neglect 


The representativeness heuristic, discussed earlier, can contribute to another cognitive error: base rate 
neglect. Base rate neglect is the error of underemphasizing the base rate—that is, the prevalence—of a phe- 
nomenon when evaluating the likelihood of an occurrence. For example, to evaluate the likelihood that a 
child has been sexually abused, a school psychologist must take into account not only the validity of the 
psychological tests administered to detect abuse, but also the base rate of abuse in his sample (see Wood, 
1996, for a user-friendly introduction). In this context, it is crucial to remember that even a test with 
extremely high validity can be virtually useless clinically when administered to detect an extremely low 
base rate phenomenon (Meehl & Rosen, 1955). Bayes theorem teaches us that all things being equal, the 
lower the base rate of a phenomenon, in this case, abuse, the lower will be the hit rate of one's predictions 
(Finn & Kamphuis, 1995). Conversely, the higher the base rate of a phenomenon, the higher will be the hit 
rate. Hence the need for valid screening tests, which can help to isolate subsamples in which the base rate 
of a disorder is high, in this way maximizing the hit rates of our predictions. 

To return to the child sexual abuse example, a school psychologist may be so impressed by the “match” 
of a child's symptoms to those of other children who have been abused (e.g., they may say to themselves, 
“Wow, this child behaves like so many of the other sexually abused children I've worked with”)—a conse- 
quence of an over-application of the representative heuristic—they may tend to neglect or ignore the base 
rate of abuse in their sample. Yet because this base rate may be extremely low, the school psychologist 
may conclude erroneously that the child was sexually abused even when she was not. Because diagnostic 
information (e.g., a child's presenting psychological symptoms) tends to be much more dramatic and salient 
than base rate information, which is often dry and abstract, it tends to be overweighed relative to such infor- 
mation (Nisbett & Ross, 1980). 


3.8. Bias blind spot 


As you read about the cognitive errors we have described, you may have thought to yourself, “Although 
this information may be important for other people, it really doesn't apply to me, because I'm not that biased.” 
If so, you're hardly alone. As Emily Pronin and others have noted (e.g., Ehrlinger, Gilovich, & Ross, 2005; 
Pronin, Gilovich, & Ross, 2004; Pronin, Kruger, Savtisky, & Ross, 2001; Pronin, Lin, & Ross, 2002), we often fail 
to recognize the existence of cognitive biases and errors in ourselves while readily detecting them in 
others. She and her colleagues argued that this bias blind spot may stem from a combination of factors, 
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including (a) naive realism (we assume that we are seeing the world as it is, so that people who disagree 
with us are surely mistaken), (b) the ubiquitous propensity to see ourselves more positively than others 
(the “Lake Wobegon effect”; see Dunning, Meyerowitz, & Holzberg, 1989; John & Robins, 1994), and (c) the 
tendency to rely more heavily on our own introspections than on others’ (Pronin et al., 2004). Although 
many school psychologists (Nelson & Machek, 2007) and similar professionals may agree that learning 
about evidence-based practices is important for their colleagues, the bias blind spot may lead them to believe 
that such knowledge is not especially important, let alone necessary, for them (as though they were saying, “I 
don't need research—I know when my treatments are working”). They may see themselves as largely immune 
to biases that afflict others; for this reason, bias blind spot is sometimes referred to as the “not me fallacy” 
(Lilienfeld et al., 2011). Nevertheless, research on the bias blind spot reminds us that we are all prone to biases 
in evaluating information under conditions of uncertainty, even as we may be oblivious to our own biases. 


4. Science vs. pseudoscience: ten warning signs 


The cognitive errors we have discussed, as well as others (see Table 1 for additional cognitive errors 
relevant to school psychology), can lead to suboptimal school psychology practices; in some cases, they 
can lead school psychologists to fall prey to pseudoscientific claims. As noted earlier, pseudosciences can be 
thought of as disciplines that display the superficial appearance of science but that lack its substance 
(Hines, 2003; Lilienfeld et al., 2003). In contrast to many other implausible claims, they can be especially 
dangerous because they so often seem to be convincing on the surface. As a consequence, they can fool 
even highly intelligent and well trained individuals (Lilienfeld et al., 2009). As is the case for cognitive 
errors, acceptance of pseudoscientific claims is rarely a matter of low overall intelligence (Hyman, 
2001; Stanovich, 2009) or insufficient education. Instead, we can all fall victim to pseudoscientific claims 
unless we are cognizant of their most common characteristics (Herbert, 2003). 

Science and pseudoscience almost surely differ from each other in degree rather than in kind, so there 
is unlikely to be a single necessary or sufficient criterion for distinguishing science from pseudoscience. 
Nevertheless, we can identify several helpful indicators or “warning signs” of pseudoscience (see also 
Bunge, 1983; Langmuir, 1953; Lilienfeld et al., 2003; Lilienfeld & Landfield, 2008; Ruscio, 2006, for these 
and other warning signs). Although none of the warning signs we discuss provides dispositive evidence 
that a claim is dubious, the more such signs a claim exhibits, the more skeptical of it we should become. 
Many dubious claims in school psychology, we should note, embody several warning signs simultaneously. 

Here we delineate 10 warning signs (see Table 2) that should be especially helpful to school psychologists 
in differentiating scientific from pseudoscientific assertions and practices. Such warning signs can come in 
handy to school psychologists when evaluating claims in the news media, on the internet, at workshops, in 
the peer-reviewed literature (but see the fourth warning sign that follows), and even in conversations with 
their colleagues. Along with each sign, we present one example from the school psychology literature (or al- 
lied literatures) that helps to illustrate it. 


Table 2 
Ten warning signs of pseudoscience relevant to school psychologists. 





Pseudoscience indicator 





Lack of falsifiability and overuse of ad hoc hypotheses 
Lack of self-correction 

Emphasis on confirmation 

Evasion of peer review 

Overreliance on testimonial and anecdotal evidence 
Absence of connectivity 

Extraordinary claims 

Ad antequitem fallacy 

Use of hypertechnical language 

Absence of boundary conditions 
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4.1. Warning sign # 1: Lack of falsifiability and overuse of ad hoc hypotheses 


According to philosopher of science Sir Karl Popper (1959), falsifiability—the capacity to disprove a 
theory with evidence—is a key characteristic of science, whereas unfalsifiability is a key characteristic of 
pseudoscience. Popper (1959) famously criticized Freud's and Adler's personality theories as unfalsifiable, 
as they were consistent with virtually any set of observed findings. For example, Adler (1922) proposed 
that the core organizing force in human personality is the striving for superiority: We desire to be better 
than others. As Popper noted, this theory is virtually impossible to falsify and therefore unscientific. A person 
who aspires to become a powerful world leader could be accommodated within the theory, but according to 
Adler, so could a person who becomes a chronic alcoholic; the latter person could merely be using his drinking 
to persuade himself that “Had I not become a heavy drinker, I surely would have become a great person.” A 
theory that explains every conceivable outcome, Popper observed incisively, in effect explains nothing. As a 
consequence, it is no longer scientific. Scientific theories must not only anticipate certain outcomes; they 
must preclude other outcomes. 

Theories can be rendered essentially unfalsifiable by the overuse of ad hoc hypotheses (Law, 2011). An 
ad hoc hypothesis is essentially an “escape hatch” or “loophole” that researchers and clinicians can use to 
explain away negative findings—results that challenge their hypotheses. For example, when a scientist 
conducts a study designed to test the hypothesis that ADHD is associated with laboratory deficits in sustained 
attention obtains negative results, she may initially doubt these results. As a consequence, she may invoke one 
or more ad hoc hypotheses to account for these unexpected findings. Perhaps the sustained attention task she 
used was inappropriate for this population; perhaps her diagnoses of ADHD were not made as rigorously as in 
previous studies; perhaps the sample size in her study was too small; perhaps the comparison group of 
non-ADHD children actually included several children with mild attentional problems; and so on. One 
or more of these ad hoc hypotheses may even be correct. 

In some cases, ad hoc hypotheses have a legitimate role to play in science, especially when they 
strengthen the theory's content, predictive power, or both (Law, 2011; Meehl, 1990). In contrast, in pseu- 
dosciences, ad hoc hypotheses rarely strengthen either, and instead are typically invoked as a desperate 
means to salvage claims from all but certain falsification. The overuse of ad hoc hypotheses can render 
claims virtually impossible to falsify in practice, because an accumulation of such hypotheses can in effect 
serve as impenetrable protective shields against negative findings. As a consequence, the overuse of ad hoc 
hypotheses can place claims outside the purview of scientific testing. 

Example: Despite overwhelming scientific evidence that demonstrates the ineffectiveness of facilitated 
communication (FC) for autism and other developmental disabilities (see Jacobson, Mulick, & Schwartz, 
1995; Mostert, 2001, 2010, for reviews), most proponents of FC continue to insist that this body of research 
is flawed. FC was designed to allow individuals with a wide range of developmental disabilities (e.g., autism, 
cerebral palsy, Down syndrome) to communicate via typewriters or keyboards with the aid of a facilitator. 
The facilitator supports a nonverbal individual's hands to allow him or her to type out meaningful messages 
(purportedly without the facilitator's influence). Nevertheless, a plethora of carefully controlled studies have 
demonstrated that facilitators are unknowingly influencing the substance of these messages via an ideomotor 
(“Ouija board”) effect (Jacobson et al., 1995; Lilienfeld, 2005). By means of the ideomotor effect, facilitators’ 
thoughts are unconsciously influencing their actions, and guiding the nonverbal individual's fingers to 
the desired keys. 

Still, FC supporters dispute this overwhelming negative verdict, often by invoking a variety of ad hoc 
explanations. Some have criticized experimental studies of FC as failing to acknowledge FC's inherent 
“complexities” (e.g., Emerson, Grayson, & Griffiths, 2001; Sheehan & Matuozzi, 1996), whereas others insist 
that evidence of FC's efficacy has not emerged because researchers give participants tasks that are too difficult 
(e.g., Borthwick & Crossley, 1993) or rely on facilitators who do not believe in the method or facilitate incor- 
rectly (Brown, 1982; Casti, 1989). Although these criticisms may initially appear legitimate, they render 
claims concerning the effectiveness of FC virtually unfalsifiable. Moreover, they fall prey to what Meehl 
(1974) called a “double standard of evidential morals” (p. 264), a tendency to shift the bar for evidence 
depending on whether it supports or disconfirms one's hypotheses. For example, the claim that FC does not 
work in certain cases because the tasks are too difficult is belied by evidence that many of the experimental 
tasks used in FC research are far simpler than those that were adduced as positive evidence of FC for the same 
children. As Jacobson et al. (1995) noted, “Controlled research that disconfirms the phenomenon is criticized 
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on grounds of rigor and procedure not even remotely addressed by studies that purport to demonstrate the 
effectiveness of the technique” (p. 757). In essence, FC becomes impossible to falsify in practice because 
proponents of the method continue to insist that they alone are qualified to test it appropriately. 


4.2. Warning sign # 2: Lack of self-correction 


Over the long haul, scientific claims tend gradually to be self-correcting, as scientific methods are 
designed to root our errors in our web of beliefs (Quine & Ullian, 1978). In contrast, pseudosciences 
tend to be stagnant over lengthy time periods, often showing scant change in the face of repeated contrary 
evidence. Ironically, science's provisional status, which has sometimes led postmodernist scholars to call it 
into question, is one of its foremost strengths, because accepted scientific knowledge is always open to revi- 
sion pending new data. At the same time, some scientific findings, including those in psychology (e.g., the 
phenomenon of classical conditioning and the bystander intervention effect), are so well established that 
we can place substantial confidence in them, even as we recognize that there is some remote possibility 
that they could be overturned or revised one day. 

Example: The idea of matching students’ learning styles to teachers' teaching styles has remained perva- 
sive in both research and practice despite a wealth of evidence pointing to its invalidity (Pashler, McDaniel, 
Rohrer, & Bjork, 2008; Stahl, 1999; Tarver & Dawson, 1978). Taking shape in the 1940s, the concept of learn- 
ing styles suggests that individuals differ significantly from one another in the way they process information. 
Undeniably, there is some truth in this notion. Nevertheless, many proponents of this concept go much 
further, arguing that to maximize students’ potential for learning, instructors must match their teaching 
methods to their students’ distinctive learning styles (Pashler et al., 2008). 

Although this “matching hypothesis” may seem intuitive, research on learning styles has not supported it. 
In a review of 15 studies that differentiated visual from auditory learners, Tarver and Dawson (1978) con- 
cluded that “modality preference and method of teaching reading do not interact significantly” (p. 25). 
More recently, in a comprehensive examination of the literature on learning styles, Pashler et al. (2008) 
found only one study that offered evidence for matching teaching styles to learning styles that met their cri- 
teria for adequate evidence (Sternberg, Grigorenko, Ferrari, & Clinkenbeard, 1999), and even this study was 
marked by serious methodological flaws. In contrast, Pashler et al. located numerous well-designed studies 
meeting their criteria that directly contradicted the matching hypothesis. For example, Massa and Mayer 
(2006) conducted a series of three computer-based studies that paired visual and verbal learners with spe- 
cially designed “help-screens” that matched their learning preferences. In general, the researchers found no 
relation between performance levels and pairing with preferred help-screens. Similarly methodologically rig- 
orous studies reviewed by Pashler et al. (e.g., Constantinidou & Baker, 2002; Cook, Thompson, Thomas, & 
Thomas, 2009) offer no support for the matching hypothesis of learning styles. 

Flouting this negative research evidence, proponents of learning styles have established a flourishing 
market for assessment and measurement tools (Pashler et al., 2008). Many educational institutions, including 
Yale University (Gawboy & Greene, 2011), continue to emphasize the importance of utilizing “different 
modes of communication to serve a range of learners” and scores of curricula are based on the largely discre- 
dited matching hypothesis (Lilienfeld et al., 2010). Given the wealth of evidence contradicting the match- 
ing hypothesis of learning styles, students and teachers would do well to turn to educational techniques 
that are grounded in more solid scientific research. 


4.3. Warning sign # 3: Emphasis on confirmation 


As we noted earlier, confirmation bias can lead us to seek out evidence that is consistent with our hypoth- 
eses and to ignore or selectively reinterpret evidence that is not. Sciences incorporate systematic safeguards 
against confirmation bias, such as randomized control designs; in contrast, pseudosciences tend to cavalierly 
flout such safeguards. Moreover, in pseudosciences, positive results are generally accorded considerably more 
weight than negative results; this emphasis on confirmation rather than refutation runs counter to the prin- 
ciple that science works best by subjecting claims to the risk of falsification (Meehl, 1978). As philosopher of 
science Mario Bunge (1967) wrote colorfully, “the pseudoscientist, like the fisherman, exaggerates his catch 
and neglects his failures or excuses them” (p. 36). 


24 S.O. Lilienfeld et al. / Journal of School Psychology 50 (2012) 7-36 


Example: School psychologists may be familiar with the lofty claims associated with the widely publicized 
“Total Transformation Program” (www.thetotaltransformation.com), which has been advertised extensively 
on television and radio commercials. As is the case with many questionable claims (see Warning sign # 7), 
James and Janet Lehman, the creators of the program, assert that it can be used to treat “every behavior 
problem.” To bolster their claims (and most relevant to the current warning sign of pseudoscience), the cre- 
ators present numerous quotes and reviews on their website as evidence of their program's effectiveness. 
Website visitors are greeted with quotations like, “My kid did a complete 180” and “The Total Transformation 
actually works as advertised,” as well as with videos of actual people providing testimonials of how the 
program changed their lives (see also Warning Sign #5 for a discussion of anecdotal and testimonial 
evidence). Nevertheless, the website contains no mention of the possibility that the program may not work 
for all children. In fact, the Lehmans go so far as to insist that they are “100% confident” that their program 
will “turn your defiant child around,” and even offer parents a full $327 refund in return for written feedback 
(as long as it is sent within 90 days of receiving the product). Curiously, however, the advertisements request 
feedback from users not to improve the program, but to help the Lehmans “prove” the product's effectiveness. 
Although a few aspects of their program may well be based on some empirically-supported behavioral 
therapy techniques (Forgatch & Patterson, 2010), it is exceedingly unlikely that “The Total Transformation,” 
even if highly effective, will work for everyone. It also seems unlikely that the Lehmans will advertise any 
negative feedback they receive on their website or on commercials. 


4.4. Warning sign # 4: Evasion of peer review 


Although the peer-review process is far from perfect (Godlee & Jefferson, 2003), and is not free of political 
influence or biases, it typically affords at least some protection against poor quality research. In the peer-re- 
view process, journal editorial boards and grant review committees send articles and grant proposals out for 
review by several, often three or more, independent (and usually anonymous) scholars. Whereas sciences 
rely on peer review as a partial safeguard against error, many pseudosciences fly under the radar of peer re- 
view, disseminating and promoting claims before they have been subjected to careful scrutiny by experts. 

Example: The popular Brain Gym program developed in the 1970s by Paul and Gail Dennison has become 
increasingly popular in educational settings. Brain Gym is based on the largely unsupported claim that learn- 
ing difficulties result from disconnections between certain areas of the brain and the rest of the body (Hyatt, 
2007). The program prescribes a series of 26 movements designed to strengthen neural pathways thought to 
be unintegrated in individuals with learning difficulties. Since its development, this program has skyrocketed 
in popularity; it is now marketed in over 80 countries (www.braingym.org/about). 

Despite the commercial hype surrounding Brain Gym, this program has consistently managed to escape 
peer scrutiny. The official Brain Gym website contains a page titled “Brain Gym Studies,” which boasts an ex- 
pansive list of investigations seemingly supporting the benefits of the program. Although this list may appear 
impressive, on closer inspection it becomes evident that most of the evidence is qualitative or anecdotal 
(www.braingym.org/studies). Of the studies that are experimental in nature, only two have been published 
in peer-reviewed journals (Khalsa, Morris & Sifft, 1988; Sifft & Khalsa, 1991). In a review of the literature, 
Hyatt (2007) identified only five peer-reviewed articles that examined the effectiveness of Brain Gym. Of 
these five, “one was discarded because the author of that article was one of the four participants in the 
study” and three others “were published in the same journal, in which authors must pay for publication” 
(Hyatt, 2007, p. 120). In an effort to explain the dearth of peer-reviewed articles examining the efficacy of 
the program, the Brain Gym website states that “Articles published in peer-reviewed journals are generally 
required to be based on scientific studies. ... For the ethical reasons mentioned above, and because of the ex- 
pertise required for statistical work and the high costs of doing such research over time, we haven't yet seen 
many such studies” (http://www.braingym.org/studies). But one cannot have things both ways: if one does 
not possess the expertise or resources to conduct studies on the effectiveness of one's program, one must 
await such studies before promoting the program widely to parents, teachers, and others. 


4.5. Warning sign # 5: Overreliance on anecdotal and testimonial evidence 


As some scholars have observed, the plural of anecdote is not fact (see Lilienfeld & Landfield, 2008). In 
other words, multiple pieces of anecdotal evidence (e.g., “My son had a severe learning disability for years; 
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now, after the school's treatment, he's cured!”) are rarely, if ever, sufficient to conclude that an assertion 
(in this, case, the assertion that the treatment was effective) is correct. This conclusion holds because 
anecdotes are (a) often difficult to verify, (b) of unclear or questionable representativeness, and (c) almost 
always vulnerable to multiple competing explanations. As Gilovich (1991) observed, the relation between 
anecdotal evidence and treatment efficacy is typically asymmetrical. If a treatment is efficacious, one should 
certainly expect to hear anecdotal claims of its effectiveness from at least some consumers. In this respect, 
anecdotes, like clinical intuition, can be helpful in the context of discovery (Reichenbach, 1938), as they 
may provide researchers with sufficient grounds for investigating a promising intervention. Yet if one 
hears anecdotal claims of a treatment's effectiveness from consumers, one should not conclude that the treat- 
ment must be efficacious. Only controlled studies that have been replicated by independent scholars can 
allow one to draw this conclusion. 

Example: Scared Straight programs rely primarily on scattered anecdotes to validate their effectiveness, 
ignoring evidence that these programs are potentially detrimental to adolescents (Lipsey, 1992; Petrosino, 
Turpin-Petrosino, & Buehler, 2003). Originating in New Jersey's Rahway State Prison in the 1970s, Scared 
Straight programs bring at-risk and delinquent adolescents to high-security prisons in an attempt to scare 
them away from future lives of crime (Lilienfeld, 2007). These programs acquired initial popularity with 
the airing of a 1979 Academy Award-winning documentary “Scared Straight!” They recaptured public atten- 
tion on January 13, 2011, when the A&E Network introduced a new series “Beyond Scared Straight”, produced 
by Arnold Shapiro, director of the original “Scared Straight!” documentary. Although Scared Straight Programs 
have gained support from policy makers and the general public (Petrosino, Turpin-Petrosino, & Flinckenauer, 
2000), research shows that this support is not buttressed by research evidence. Program evaluations and sev- 
eral randomized studies demonstrate that Scared Straight programs are ineffective in reducing crime rates 
among adolescents (e.g., Finckenauer, 1982; Lewis, 1983; Yarborough, 1979; see Petrosino et al., 
2000, for a discussion). In fact, in a meta-analysis of seven quasi-experimental studies, Petrosino 
et al. (2003) showed that involvement in Scared Straight programs significantly increased the risk 
of offending among adolescents (see also Lilienfeld, 2007). 

Despite this negative evidence, proponents of Scared Straight programs continue to invoke anecdotal 
reports and testimonials as evidence for their effectiveness. For example, in a recent interview, producer 
Arnold Shapiro stated that he is convinced that Scared Straight programs are effective because of what 
“youth counselors and teachers and police officers and others who bring kids to these programs tell 
me.” He went on to claim that there are no negative consequences of Scared Straight because in speaking 
with the original participants, “14 out of the 17 attributed their turn-around to that prison visit” (Sharenow 
& Shapiro, 2011). Although individual success stories can be moving, they cannot provide scientific evidence 
for the efficacy of a practice. Among other things, informal self-reports of improvement are susceptible to a 
host of alternative explanations, including regression to the mean, placebo effects, demand characteristics, 
retrospective biases, and effort justification (i.e., the tendency to justify the time, energy, and expense of a 
treatment by convincing oneself that it must have worked; see Lilienfeld et al., 2008, for a discussion of 
these and other ways in which people can be fooled by ineffective or harmful therapies). Instead, lawmakers, 
clinicians, and film-makers alike must turn to well-controlled data to establish the efficacy of their programs. 


4.6. Warning sign # 6: Absence of connectivity 


Keith Stanovich (2009) defined “connectivity” as the extent to which claims build on, or “connect up 
with,” other claims. Developed sciences tend to display a cumulative character, with novel claims drawing 
on well-established findings. In contrast, pseudosciences often lack connectivity with other knowledge, 
purporting to construct entirely new paradigms out of thin air (see Herbert et al., 2000, for a striking example 
within clinical psychology). Of course, in rare cases, entirely new paradigms that overthrow extant knowl- 
edge emerge in science, a phenomenon that Kuhn (1962) famously termed “revolutionary science.” Yet 
such spectacular successes are exceedingly few and far between, and are accompanied by especially persua- 
sive evidence that the new paradigm can predict findings than previous paradigms cannot (see also 
“Extraordinary claims” section that follows). 

Example: Proponents of the idea that some individuals are “left-brained” whereas others are “right- 
brained” draw on legitimate scientific findings indicating that the two hemispheres often differ somewhat 
in their cognitive styles (Sperry, 1974). Nevertheless, these advocates ignore the sizeable corpus of 
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research showing that both hemispheres are in virtually continuous interaction and do not inhabit separate 
psychological “worlds” (Corballis, 1999; Hines, 1987). They also ignore evidence that the two hemispheres 
are much more similar than different in their information processing. 

According to advocates of the extreme laterality view, “left-brained” individuals are more logical and 
analytic than right-brainers, whereas “right-brained” individuals are more creative, emotional, and artistic 
than left-brainers. In fact, the differences between these two groups of individuals are only of degree, not 
of kind; moreover, being logical and analytic is not mutually exclusive with being creative, emotional, and 
artistic. Erroneous beliefs of this nature can influence how student learning and activities are structured, 
with seeming left-brainers afforded more opportunities to practice logic skills, and seeming right-brainers 
afforded more opportunities to be creative and artistic. For examples, numerous websites present different 
school curricula or teaching methods for “left-brain” versus “right-brain” learners (e.g., http://www. 
homeschool-by-design.com/right-brain-vs-left-brain.html). One contends that “yucky, bulky textbooks” 
are for left-brain learners and that right-brain learners require visual aids for learning (http://www. 
study-skills-for-all-ages.com/right-brain-left-brain.html; see also Warning sign # 2 for a discussion of 
misconceptions regarding student learning styles). An online search of the terms “left-brain” or “right- 
brain” also yields numerous quizzes and blogs that help parents and teachers determine which side of a 
child's brain is dominant (e.g., http://www2.scholastic.com/browse/article.jsp?id=3629). Apparently, 
believers in the myth of left brainers and right brainers are content to remain disconnected from the 
broader scientific literature. 


4.7. Warning sign # 7: Extraordinary claims 


As Marcello Truzzi (1978) and later Sagan (1995a,b) argued, extraordinary claims require extraordinary 
evidence. That is, if one advances an assertion that runs counter to virtually all existing knowledge, one 
must provide especially convincing evidence for this assertion. This principle dovetails with the basic 
principles of Bayesian thinking, an approach that incorporates the a priori plausibility of assertions 
when evaluating their evidentiary strength (see Goodman, 1999; Lilienfeld, 2011). Yet many pseudosciences 
advance extraordinary claims without offering equally extraordinary evidence. 

Example: In the field of school psychology and allied fields, the concept of “Indigo Children” (Carroll & 
Tober, 1999, 2009; Tappe, 1999) is a prime example of an extraordinary claim that is not accompanied by 
equally extraordinary evidence. Two proponents of the concept, Carroll and Tober (1999), wrote that: 


An Indigo Child is one who displays a new and unusual set of psychological attributes and shows a 
pattern of behavior generally undocumented before. This pattern has common unique factors that 
suggest that those who interact with them (parents, in particular) change their treatment and up- 
bringing of them in order to achieve balance. To ignore these new patterns is to potentially create 
imbalance and frustration in the mind of this precious new life (p. 1). 


During a 2009 interview on the “Virtual Light Broadcast” (http://www.lightworker.com/VirtualLight/ 
VLBroadcasts/2008/2008_12_20.php), Lee Carroll similarly claimed that Indigo Children possess a “different 
kind of consciousness” that allows them to intuitively “know how everything works.” He suggested that 
Indigo Children know things that their parents do not and are privy to information that is beyond most 
peoples' comprehension. Carroll and others (e.g., Tappe, 1999) believe that most individuals born in 
the late 1960s and beyond can be characterized as Indigos and that they emit an Indigo “energy color” 
(which can be perceived only by special people). Many proponents of the concept of Indigo Children further 
believe that the prevalence of ADHD has increased in recent years (Khan & Faraone, 2006) because attentional 
problems are a byproduct of the highly evolved consciousness that Indigos are said to possess. 

Although claims about Indigo Children are expansive, there is no convincing or even suggestive evidence 
that humans born during the past few decades are somehow special. Proponents of the concept typically only 
offer anecdotal evidence to buttress their claims (see Warning sign # 5). In fact, when one looks at the “ten 
most common traits of Indigo children” (Carroll & Tober, 1999, p. 1), the descriptions seem more consistent 
with the possible increase in narcissism among recent generations (Twenge & Foster, 2008; but see Donnellan 
& Trzesniewski, 2009, for a different view), rather than the sudden emergence of a special class of human 
beings equipped with remarkable cognitive abilities. For example, although children today may indeed be 
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more likely to “get frustrated with systems that are ritual-oriented and don't require creative thought” 
(Carroll & Tober, 1999, p. 2), this does not mean that they herald the next great leap in human evolution. 
It is far more plausible that the increased rates of the ADHD diagnosis in recent years are due to greater rec- 
ognition and sensitivity to the features of the disorder (or to what epidemiologists call “detection bias”) than 
to Indigo children's ostensible special talent for “see[ing] better ways of doing things” (Carroll & Tober, 1999, 
p2): 


4.8. Warning sign # 8: Ad antequitem fallacy 


If a practice has been popular for a long time, it may be tempting to assume that there must be something 
to it. Yet there are glaring counterexamples to this conclusion: Astrology has been used for about 4500 years 
despite a decided absence of evidence for its validity (Hines, 2003). The problem here is that practices may be 
enduring not because they are effective, but because certain cognitive biases, like confirmation bias and 
illusory correlation, have consistently fooled people for decades or even centuries into believing that they 
are effective. Pseudoscientific disciplines often fall prey to the ad antequitem fallacy, the error of invoking 
the antiquity of claims as a means of bolstering their legitimacy. 

Example: As suggested earlier, one of the most problematic trends in school psychology has been the 
continued uncritical use of certain projective methods, such as human figure drawing tests and many 
Rorschach Inkblot Test indices, in the absence of convincing support for their validity (Gresham, 1993; 
Hojnoski et al., 2006; Miller & Nickerson, 2006). Some have speculated that these methods owe their lon- 
gevity to a combination of cognitive biases and “institutional inertia”—the tendency of longstanding clin- 
ical traditions to change slowly (Lilienfeld, Wood, & Garb, 2007). Hojnoski et al. (2006) similarly 
suggested that the continued inappropriate use of projective techniques may stem from “simple habit” 
(p. 156). 

Proponents of projective techniques, however, have sometimes argued that their continued use bolsters 
their credibility. For example, when arguing for the validity of the Rorschach Inkblot Test, Weiner, Spielberger, 
and Abeles (2002) asked rhetorically: 


How likely is it that so many Rorschach assessors have been using the instrument for so long, in so 
many places and contexts, solely on the basis of illusory correlation? If this seems unlikely, is it unrea- 
sonable to infer that there has been some utility in their work? (p. 11) 


Although the questions posed by Weiner and colleagues are worth asking (and indeed, at least some 
Rorschach indices, especially those designed to detect thought disorder, possess at least some degree of 
validity; see Lilienfeld et al., 2000), they neglect the possibility that erroneous conclusions can be passed 
on to subsequent generations of professionals. In an effort to understand the enduring popularity of projective 
methods, Hojnoski et al. wondered whether “the training one receives may highly influence choice of 
assessment methods” (p. 156). The words of counseling psychologist Donald Super are also germane in 
this context: “Unorganized experience, unanalyzed data, and tradition are often misleading. Validation should 
be accomplished by evidence gathered in controlled investigations and analyzed objectively, not by the 
opinions of authorities and impressions of observers” (see Buros, 1949, p. 167). In other words, tradition 
should not be confused with validation. 


4.9. Warning sign # 9: Use of hypertechnical language 


In an effort to accurately differentiate among superficially similar concepts, scientists often find it useful to 
introduce new and distinct terminology. Nevertheless, this practice can be abused, especially when the 
terminology provides unsupported techniques with a cachet of unearned scientific respectability. Promoters 
of pseudoscience often rely on overly technical language in an effort to establish legitimacy for their otherwise 
invalid practices. In contrast to the language of well-established sciences, this language frequently lacks 
precision, meaning, or both (Lilienfeld & Landfield, 2008). 

Example: Proponents of Auditory Integration Training (AIT) routinely use hypertechnical language 
when explaining the purported effectiveness of this practice for autism spectrum disorders. Developed by 
French otolaryngologist Guy Berard, AIT generally consists of two daily half-hour sessions for approximately 
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10 days in which affected individuals listen to music that varies in volume and pitch (Herbert, Sharp, & 
Gaudiano, 2002). Proponents of AIT maintain that autism involves a hypersensitivity to sound, and 
that auditory intervention allows for the retraining of a disordered auditory system (AIT Institute: Audi- 
tory Integration Training). Although these claims sound impressive, research over the past three decades 
has yielded negative evidence for the effectiveness of AIT for autism spectrum disorders (Bettison, 1996; 
Gillberg, Johansson, Steffenburg, & Berlin, 1997; Mudford, et al., 2000; Zollweg, Palm, & Vance, 1997). 

Despite this absence of supportive evidence, AIT has acquired substantial popularity as a treatment for 
autism spectrum disorders. This popularity may be buttressed in part by the rampant use of hypertechnical 
language in expositions of this treatment. For example, on the AIT Institute Website, proponents of this 
practice claim that it “efficiently retrains a disorganized auditory system and improves hearing distortions 
and sound sensitivity” (www.aitinstitute.org). Similarly, the Berard AIT website explains that “The Berard 
method of auditory integration training stimulates the auditory system with unique sounds [that] stimulate 
the auditory system to reduce or eliminate the problems with this system” (http://www.berardaitwebsite. 
com/WhyBerard.htm). Still another website says that “The Kirby Method of AIT uses classical music exclu- 
sively (not the pop music used by Berard practitioners). Classicfal (sic) music is used because of the statistical 
content that mirrors normal brain function (theorizes Professor Anderson of Harvard). The active modulation 
is determined by the power density spectrum of the music.” (http://www.kirbyait.com/). Although these 
explanations of AIT as a treatment for autism spectrum disorders may sound legitimate, they are vague and 
lack clear-cut scientific referents. 


4.10. Warning sign # 10: Absence of boundary conditions 


Scientific theories and practices typically incorporate clearly defined boundary conditions: instances in 
which the claims associated with the theory are, and are not, applicable. For example, although there is 
clear support for the efficacy of exposure therapies—those that systematically expose individuals to 
anxiety-provoking stimuli, often for prolonged time periods—for anxiety disorders (Barlow, Allen, & Basden, 
2007), no credible proponents have proposed that they are efficacious in the treatment of schizophrenia or 
autism, let alone Alzheimer's disease or cancer. In stark contrast, “most pseudoscientific phenomena 
are purported to operate across an exceedingly wide range of conditions” (Lilienfeld et al., 2003; p. 9; see 
also Hines, 2003), such that proponents may claim that their concept or treatment applies to virtually all 
conditions, or virtually everyone. 

Example: The Irlen Method, devised by Licensed Marriage and Family Therapist Helen Irlen, is character- 
ized by a conspicuous absence of boundary conditions. In 1983, Irlen claimed to have discovered a previously 
unknown condition called Scotopic Sensitivity Syndrome, which supposedly results in problems with 
reading, behavior, and emotion (Hyatt, Stephenson, & Carter, 2009). Irlen asserted that this syndrome 
could be treated only using colored lenses and overlays, which “improve the brain's ability to process visual 
information” (www.irlen.com). Although Irlen's claims are bold, neither she nor her supporters have 
provided adequate evidence to support them over the past three decades (Evans & Drasdo, 1991; Hyatt 
et al., 2009). 

Yet Irlen continues to advertise her Irlen lenses as an effective treatment for a plethora of psychological 
and physical conditions. On the Irlen Method Website, Irlen claims that her “non-invasive technology ... 
can improve reading fluency, comfort comprehension, attention and concentration while reducing light 
sensitivity” (www.irlen.com). She goes on to claim that Irlen colored overlays and filters can help individuals 
suffering from “Reading and learning problems, Dyslexia, ADD/HD, Autism, Asperger Syndrome, behavioral 
and emotional problems, headaches, migraines, fatigue and other physical symptoms, light sensitivity/ 
photophobia, Traumatic brain injury, whip lash, concussions, and certain medical and visual conditions” 
(www.irlen.com). Indeed, it is difficult to conceive of any psychological condition for which Irlen lenses 
would not purportedly be effective. Irlen's expansive claims clearly lack boundary conditions: she puts 
forth her method as a cure-all treatment for virtually all disordered individuals. 


5. Parting words, implications, and prescriptions 


In this article, we have offered a user-friendly primer of scientific thinking principles for school psycholo- 
gists and a set of guidelines for distinguishing scientific from pseudoscientific claims. We suggest that in an 
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ideal world, these principles should come to take their place alongside of the WISC-IV and Woodcock- 
Johnson (Wechsler, 2004; Woodcock, McGrew, & Schrank, 2007), and other standard psychometric tools 
as essential components of the school psychologist's everyday armamentarium. More broadly, we have 
argued that scientific thinking skills, which we can think of as tools designed to compensate for cognitive 
biases to which all humans are prone, are ultimately school psychologists’ best safeguards against errors 
in judgment and decision-making. In turn, they are also school psychologists’ optimal protection against 
the seductive charms of pseudoscientific practices and myths, many of which seem persuasive at first 
blush but lack compelling scientific support. 


5.1. The limitations and powers of science 


Of course, science and scientific thinking, being inherently human products, are themselves fallible and 
prone to error. The scientific process is not immune to biases, political influence, or in rare cases, outright 
fraud. Yet most of the shortcomings of scientific methodology have, perhaps not surprisingly, been uncovered 
by scientists themselves. Indeed, to the extent that scientific methods are not working as efficiently or accu- 
rately as they should, scientific methods themselves can help us make them work better. For example, recent 
discussions in the scientific community and popular press have highlighted the fact that the effect sizes of ini- 
tial studies are often larger than those of later studies—the so-called “decline effect” or “law of initial results” 
(Ioannidis, Ntzani, Trikalinos, & Contopoulos-loannidis, 2001; Lehrer, 2010). That is, early spectacular findings 
in science, including psychology, are often difficult to replicate (see also Lykken, 1968, 1991). To the extent 
that the decline effect is genuine, it will be ultimately be recognized as a biasing factor in scientific studies, 
and scientists will almost certainly find ways of correcting statistically for its influence in their analyses. 

As many philosophers of science have pointed out, science, despite its inevitable imperfections, is un- 
rivaled in human history for its problem-solving power (O'Donohue & Lilienfeld, 2007). It derives much of 
this power by rooting out errors in our web of beliefs, thereby increasing the probability of—although not 
ensuring—reasonably accurate conclusions. Moreover, science, in contrast to other ways of knowing, such 
as intuition, has the virtue of being self-correcting (Thagard, 1978), even if such self-correction sometimes 
takes longer than we might wish. Perhaps more than anything else, science is a prescription for humility 
(McFall, 1996), as it is an explicit admission that we might be wrong and that we need to rely on formal 
protections—namely, scientific methods—against our own fallibility. As Tavris and Aronson (2007) noted, 
science is a method of “arrogance control”: Whether we like it or not, it forces us to doubt our cherished 
and strongly held assumptions. 


5.2. Educational implications 


If our position has merit, it suggests several educational implications for the training of school psycholo- 
gists. All graduate students in school psychology, we believe, must learn about the cognitive errors to 
which all humans are susceptible, and come to understand why scientific methods, such as randomized 
controlled trials, are essential bulwarks against these errors. Equally essential, school psychology students 
must come to understand that their discipline has evolved over time by eliminating errors, and come to see 
scientific methods as invaluable tools for improving the rigor of their clinical judgments and research 
endeavors. These basic principles of scientific thinking should, we argue, be taught not in a single course but 
rather integrated through school psychologists’ didactic and practicum training. In addition, graduate 
students in school psychology should be exposed to key warning signs of pseudoscience so that they can 
better differentiate well-supported from poorly supported techniques and claims. 

Most of all, school psychologists in training must be encouraged to cultivate the healthy skepticism 
about which Sagan (1995a,b) wrote so eloquently: a skepticism that requires us to keep an open mind 
to novel claims while insisting on rigorous evidence. This delicate balancing act is often cognitively and 
emotionally difficult, because it requires us to avoid the extremes of excessive gullibility, one the one 
hand, and cynicism, on the other. But in the long run, this is the middle ground that school psychologists 
must seek out and occupy to evaluate the validity and efficacy of their interventions, and that will ulti- 
mately be the most beneficial for the children they serve. 
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5.3. Ten prescriptions for school psychologists 


In closing, we offer readers the following 10 bottom-line prescriptions to take with them in their school 
psychology work. These 10 prescriptions, which distill many of the core principles we have reviewed, 
should be helpful to all school psychologists, regardless of whether their principal setting is research, practice, 
or both (see Table 3). 


(1) 


Table 3 


Seek out disconfirming evidence. To minimize confirmation bias, school psychologists should actively 
search for data that might prove their hunches and hypotheses wrong. They should also be certain not 
to dismiss too quickly or readily research or clinical data that run contrary to their hypotheses; instead, 
they should seriously entertain these hypotheses at least briefly. 

Do not become overly attached to one's hypotheses. Psychologist Edward Bradford Titchener implored 
others to “Know all theories, love some, wed none.” If we become overly wedded to our hypotheses, 
we can refuse to consider evidence that could challenge our most deeply held assumptions, whether 
they be assumptions regarding a favored theoretical model or assumptions regarding the causes of a 
child's behavioral or learning difficulties. As Greenwald, Pratkanis, Leippe, and Baumgardner (1986) 
pointed out, “Theory obstructs research progress when the researcher is an ego-invested advocate of 
the theory and may be willing to persevere indefinitely in the face of prediction-disconfirming results” 
(p. 227). 

Consider rival hypotheses. As a corollary of the immediately preceding prescription, school psycholo- 
gists should be sure to ask themselves whether alternative hypotheses could account for their research 
of clinical findings just as well as their favored hypothesis. Indeed, as rule of thumb, school psycholo- 
gists should be reluctant to accept any explanation as adequately corroborated unless it has “beaten 
out” at least one other plausible rival explanation. 

Do not cherry-pick. The too-often-ignored rule of total evidence (Carnap, 1962) reminds school 
psychologists to examine all pertinent evidence, not just selected evidence, before concluding that 
their preferred hypothesis is well confirmed. Similarly, this rule cautions us to avoid cherry-picking 
evidence that conveniently fits our research or clinical hypotheses. A successful hypothesis must 
explain all relevant data, not merely data that are favorable to our position. 

Put one's intuitions to systematic tests. As noted earlier, intuitions can be helpful in the context of 
discovery (Reichenbach, 1938) but are rarely well suited to testing hypotheses systematically. We 
encourage school psychologists to routinely place their clinical intuitions to the test in “action re- 
search” (Mills, 2000), examining whether their gut hunches are correct or mistaken. 

Be skeptical of clinical wisdom. The reliance on established clinical authority—sometimes 
humorously termed “eminence based practice” as opposed to “evidence-based practice” (Isaacs & 
Fitzgerald, 1999)—can be dangerous, because longstanding clinical wisdom is not always correct. 
One vital function of school psychology research is distinguishing which aspects of clinical wisdom 
are worth retaining and which are worth discarding. 

Be cognizant of one's blind spots. Because of bias blind spot (Pronin et al., 2002), we are virtually all 
oblivious to our biases, including confirmation bias. One reason that “hermit scientists” (Gardner, 
1957) are so prone to spectacularly wrong-headed ideas is that they insulate themselves from external 
criticism, so that they never become aware of their errors in thinking. Routinely running our research 


Ten take-home prescriptions for improving school psychology research and practice. 





Seek out disconfirming evidence 

Do not become overly attached to one's hypotheses 
Consider rival hypotheses 

Do not cherry-pick 

Put one's intuitions to systematic tests 

Be skeptical of clinical wisdom 

Be cognizant of one's blind spots 

Encourage dissent 

Quantify, quantify, quantify 

Maintain a self-critical attitude 
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and clinical hypotheses past our school psychology colleagues and students—and asking them to 
detect any weak spots in these hypotheses—may be a partial antidote to serious blind spots in our 
thinking. As we have noted earlier, intelligence offers little or no immunity against bias. 

Encourage dissent. In research teams and clinical decision-making teams, we should go out of our 
way to encourage dissenting points of view. For example, we should reinforce colleagues and research 
assistants for raising alternative interpretations of findings and reinforce clinical colleagues who offer 
case conceptualizations that differ from ours. Such disagreement can be ego-bruising in the short term, 
but it will tend to result in more accurate interpretations of research and clinical data in the long-term. 
Quantify, quantify, quantify. Even those school psychologists who are math-phobic should understand 
that quantification is essential, as it can reduce uncertainty in inferences (Sechrest, 2005). For example, 
researchers who suspect that their interview-based impressions of schoolchildren possess useful 
information should attempt to assess these impressions numerically, and clinicians who want to find 
out whether their school-based interventions are efficacious should measure their client outcomes 
systematically using well-validated scales. 

Maintain a self-critical attitude. Finally, as we have noted, humility is the heart of the scientific attitude, 
which entails a willingness to acknowledge that one might be mistaken (Sagan, 1995a,b). The best 
school psychology researchers and practitioners are continually questioning their preconceptions, 
subjecting them to searching criticism, and most important, modifying their beliefs when the evidence 
compels it. 
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